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5 Claim of Priority 

This patent application claims the benefit of U.S. Provisional Application No. 
60/149,796 filed August 19, 1999, and U.S. AppUcation Nos. 09/495,971, 09/496,364, 
09/496,533, 09/496,607, filed Febraary 2, 2000, which are hereby incorporated herein by 
reference in their entirety. 

10 Background of the Invention 

Field of the Invention 

The present invention relates to imaging devices and, more particularly, to 
methods and systems for compensating for defective pixels and pixel values. 
Description of the Related Art 

15 Conventional integrated circuit imaging devices include an array of hght 

detecting elements or pixels which are interconnected to generate an analog signal 
representation of an image illuminating the device. Two common examples of 
conventional integrated circuit imaging devices are a charge coupled device (CCD) and 
a complementary metal oxide semiconductor (CMOS) image sensing device. 

20 Conventional imaging devices typically include one or more light detecting elements 

and charge storage elements. Such integrated circuits employ a hght detecting element, 
such as a photodiode or phototransistor, that is capable of conducting current 
approximately proportional to the intensity of the light illuminating the element and a 
charge storage element. The collection of the signals from the pixels represents the 

25 image of the scene viewed by the array. 

In order to produce a color image, the imaging devices separate the light into 
various color components by filtering the light before the light strikes the light detecting 
elements. The array of light detecting elements is often deposited with a filter layer 
such that neighboring pixels may have different color filters and organized in a 

3 0 particular pattern. 



Because each pixel is typically only capable of detecting a single color, 
conventional imaging devices utilize a process by which all of the color components are 
reconstructed for each pixel in order to maintain the original unfiltered array resolution. 
To reconstruct the color components, conventional imaging devices use a process of 
5 color interpolation that is performed after an analog signal associated with each pixel 

has been digitized. The conventional process of color interpolation performed after an 
analog signal associated with each pixel has been digitized requires conversion fi-om 
analog to digital (A/D) and may require extensive computations in order to achieve a 
high quality color presentation of the image. The A/D conversion and extensive 
10 computations may require hardware, such as analog-to-digital (A/D) converters, 
memory, processors and software. The hardware and software may add to the 
complexity, size and expense of the imaging device and reduce the speed of the imaging 
process. 

Each pixel represents a sample of the scene and hence is a data value in the two- 
15 dimension image produced by the imaging system. Defective pixels, commonly 

referred to as 'bad pixels,' are caused by an array defect and do not provide the correct 
hght intensity value. Bad pixels appear as image artifacts that can reduce the image 
quality significantly. In particular, a bad pixel produces an output signal that 
significantly deviates fi'om the mean output level of adjacent pixels when the exposure 
20 level of all pixels is unified. Pixels that are significantly brighter than adjacent pixels in 
a unified dark fi*ame are commonly referred to as a 'hot pixels,' while pixels that are 
significantly darker than adjacent pixel in a unified bright firame are commonly referred 
to as 'dead pixels.' 

The defective pixels are typically distributed in a random manner. However, a 
25 'bad column' (i.e., a complete column is defective) or a blemish (i.e., a cluster of 

neighboring pixels is defective) may occur and are typically discarded by the 
manufacturer. Sensor arrays that contain random defective pixels in an amount that 
does not exceed a given limit are released, and the remaining bad pixels are sometimes 
be corrected in some other manner. 
30 Some CCD and CMOS integrated circuit color imaging devices employ a 

process of bad pixel detection and correction. Conventionally, the bad pixel detection 
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step is performed off-line by the manufacturer, before the imaging device is shipped. A 
bad pixel list is typically stored in an EEPROM lookup table. When the imaging device 
is later used in a product, such a camera, the lookup table is read to locate the bad 
pixels. Once the bad pixel locations are read, the values of the bad pixels are discarded 
5 when constructing an image based upon the imaging device readout. The bad pixel 
correction is performed by substituting the bad pixel value with the value of a pre- 
specified other pixel, wherein a lookup table created during the manufacturing process 
is used to identify the pre-specified other pixel whose value is to be used. This 
conventional correction step is typically performed after the analog signal for each pixel 

1 0 has been digitized. 

The conventional bad pixel detection and correction process described has 
several significant drawbacks. For example, some of the array defects that cause bad 
pixels are temperature and gain dependent, and thus may appear and disappear during 
operation. Because these bad pixels are not consistently "bad," they may not be 

15 detected by the manufacturer, and hence, may not be corrected. In addition, because 

the detection step is performed off-line by the manufacturer, a non-volatile storage 
device is needed to store the defect pixel Ust. Adding a storage unit disadvantageous^ 
increases the cost and complexity of the system. 

20 Summarv of the Invention 

The present invention relates to methods and systems for the detection of 
defective pixels within an array of sensor cells and for correction or compensating for 
the defective pixels. The effect these defective pixels have on image quality is thereby 
reduced. In particular, one embodiment of the integrated circuit imaging system of the 

25 present invention includes an on-line, on the fly defective pixel detection and correction 
process that provides high speed operation, reduced storage requirements, and improved 
detection of defective pixels as compared to conventional systems. 

For example, one embodiment of the present invention provides an on-line bad 
pixel detection and correction process that compares a first pixel readout value with the 

30 readout values of other pixels in first pixel's local neighborhood. The average, 

weighted mean or median value of the readout values of the other pixels may be 
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calculated. The calculated value may also include the first pixel readout value. This 
calculated value may be used in the comparison with the jfirst pixel readout value. 
When the first pixel readout value varies by more than a first amoxmt as compared with 
the value calculated based on the readout values of the neighboring pixels, a value 
5 related to the readout values of the neighboring pixels is used in place of the first pixel 

readout value. 

One significant advantage of the present invention is that the correction or 
compensation for some or all bad pixels may be performed, including bad pixels and 
bad pixel values that are introduced during the real-time operation of the imaging 

10 system due to temperature and/or gain changes. Thus, bad pixels and bad pixel values 
may be detected while the imaging device is used on-line by a consumer or other end- 
user, such as in a camera or the like. By contrast, conventional techniques only correct 
bad pixels that were identified during the manufacturing process, before the imaging 
device is deUvered to the end-user. 

15 Another significant advantage of the present system is that it prevents the loss of 

edges in a reconstructed image. Conventional processes may destroy edges via the 
correction process since often times the correction involves a low pass filter that reduces 
the high frequency content of the reconstructed image. 

Still another advantage of the present system is that, because the present 

20 invention corrects bad pixels on-hne, the need for bad pixel look up tables is reduced or 
eliminated. Thus, the cost and complexity associated with non- volatile storage units 
conventional used to store bad pixel lookup tables, such as those used by conventional 
systems is reduced or eliminated. Also, the system is advantageously able to correct 
individual bad pixels distributed randomly in an array, rather than solely a cluster or 

25 colunm of bad pixels. 

In one embodiment, the imaging system includes an array of pixel sensor cells 
arranged in rows and columns, a plurality of detection circuits, and an array controller 
with a programmable readout mode. The system may comprise a monochrome or a 
color imaging system. If the imaging system is a color imaging system, then a color 

30 filter layer comprised of several color filter components organized in a predefined 
pattern may be deposited on or otherwise placed over the pixel sensor array. For 
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example, in one embodiment, the color filter comprises a primary color system with red, 
green, and blue filters (RGB) or a complementary color system arranged in a Bayer 
pattern. 

In one embodiment, the imaging system may fiirther include ampUfication 
5 stages, analog-to-digital conversion units, memory units and various other signal 

processing blocks either 'on-chip,' that is, on the same chip as the imaging device, such 
as on the same chip as a CMOS integrated circuit imaging device, or 'off-chip,' that is 
on a separate chip or circuit board than the imaging device. 

The imaging system may be a monochrome imaging system employing a CMOS 

10 integrated circuit, or may be color imaging system employed, by way of example, a 

CCD imaging device with appropriate color filters. In one embodiment, a signal 
processing circuit is used to detect and correct defective or bad pixels. The signal 
processing circuit treats bad pixels as random shot noise and utilizes non-linear 
conditional fihering for detecting and eliminating or excluding the bad or noisy pixels. 

15 In one embodiment, the color imaging system employs an on-the-fly bad pixel 

detection and correction process using a signal processing procedure performed during 
readout of the imaging device output signals, while those signals are still in analog 
form. This on-the-fly bad pixel detection and correction process is capable of random 
access readout and includes programmable gain amplifiers, an A/D converter, one or 

20 more registers for temporary storage, and a signal processing block. In this on-the-fly 

mode of operation, the signal processing unit performs the detection step via horizontal, 
vertical and diagonal conditional median filters. If the pixel whose value is being 
examined, which may be the middle or center pixel within a horizontal, vertical or 
diagonal set of pixels, the is not within the condition limits, its value is replaced in the 

25 correction step. The replacement value may be determined by the directional median 

that possesses the minimum variation. The value chosen can be, by way of example, a 
median, weighted mean, or average value of two or more pixels in the neighborhood in 
which the bad pixel is situated. 

In one embodiment, the imaging system uses random access readout. However 

30 the imaging system may also use a pipeline architecture, a parallel readout architecture, 
or other architectures. A pipeline architecture may use a sequential readout via row and 
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column shift registers and several line storage units, such as random access memory 
(RAM) in digital domain or capacitors in analog domain. In a parallel readout 
architecture, several pixels per column can be read simultaneously via independent 
column buses and stored in registers. 
5 In still other embodiments, the imaging system further combines the on-the-fly 

bad pixel detection & correction with various readout modes such as windowing and 
sub-sampling. In one embodiment, the imaging system uses additional signal 
processing such as on-the-fly color interpolation. 

The pixels of the imaging system are not required to be organized in a 

10 rectangular matrix. A similar implementation can provide on-the-fly bad pixel 
correction and detection for color imaging system and/or different pixel topography via 
modification to the readout control, the pixel neighborhood size and configuration. For 
example, a color imaging system with an RGB Bayer pattern may use a 5 x 5 pixel 
neighborhood out of which pixels of the same color component as the pixel who value 

1 5 is being examined are selected for the filtering process. 

Brief Description of the Drawings 
Figure 1 illustrates one embodiment of an imaging system coupled to a 
television. 

20 Figure 2 illustrates one embodiment of an imaging system coupled to a 

computer. 

Figure 3 illustrates a Bayer pattem color filter for the primary color system. 

Figure 4 illustrates one embodiment of a CMOS integrated circuit, monochrome 
imaging system that supports on-the-fly bad pixel detection & correction. 
25 Figures 5 A-B illustrate one embodiment of an on-the-fly bad pixel detection and 

correction process implemented by the circuit of Figure 4. 

Figure 6 illustrates an exemphfying 3x3 block of pixels within the system of 
Figure 4. 

Figure 7 illustrates an exemplifying 5x5 block of pixels using the primary color 
30 scheme and the Bayer color pattem where the middle pixel is red. 
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Figure 8 illustrates an exemplifying 5x5 block of pixels using the primary color 
scheme and the Bayer color pattern, where the middle pixel is blue. 

Figure 9 illustrates an exemplifying circuit used to locate a minimum pixel 
value, an intermediate pixel value, and a maximum pixel value in a group of three 
5 pixels. 

Detailed Description of Preferred Embodiments 
The present invention relates to a novel imaging system that provides flexible 
addressing and processing of imaging pixel sensor elements. The novel architecture of 
the present invention allows for a highly integrated, low cost imager with high speed 

10 performance and good image quaUty. Defective or bad pixels within an array of active 
or passive pixel sensor cells are detected and compensated for, thereby reducing or 
eliminating the visible effects the bad pixels may have on resulting images. In one 
embodiment, the imaging system provides an on-line, on-the-fly bad pixel detection and 
correction process that provides high speed operation, reduced non-volatile storage 

1 5 requirements, and good performance. 

For example, one embodiment of the present invention provides an on-line bad 
pixel detection and correction process and circuit that compares a first pixel readout 
value with another value derived at least in part on the readout values of other pixels in 
first pixel's local neighborhood. Based on this comparison, a determination is made as 

20 to whether the first pixel readout value is valid. If it is determined that the first pixel 

readout is not valid, a value related to the readout values of the neighboring pixels is 
used in place of the first pixel readout value, and is sent downstream as part of the 
image data to be stored, processed, displayed and/or printed. Otherwise, the first pixel 
readout value may be used. The term pixel value, as used herein, may be an analog or 

25 digital readout value, a digitized version of the analog readout value, or other 

transformed pixel readout value. 

The exemplifying imaging systems described below use a monochrome CMOS 
integrated circuit, an array of pixels organized in a rectangular matrix, and, in a color 
version, a color filter with a primary color system (RGB) in a Bayer color pattem. 

30 However, other embodiments use a charge coupled device (CCD) or may use other 

imaging technologies. Likewise, the imaging systems of the present invention may be 
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implemented with another color system, such as the comphmentary color system 
(Yellow, Cyan and Magenta) and/or another color pattern. In addition, the imaging 
system of the present invention may be implemented with the pixels organized in 
another pixel matrix or pixel topography. For example, the array does not have to be 
5 rectangular or have a regular pattem of pixels. 

The imaging system may also include either on-chip or off-chip amplification 
stages, analog-to-digital conversion units, memory units and various other signal 
processing blocks. In addition, the system may further comprise a micro-lenses layer. 
For example, in one embodiment, the bad pixel detection and correction system resides 

10 with a CMOS sensor array on the same chip or substrate, such as in a CMOS integrated 
circuit color imaging device. In another embodiment, the bad pixel detection and 
correction system resides in a separate companion chip and is used in conjunction with 
a CCD sensor array located on a separate substrate. 

Figures 1 and 2 illustrate exemplifying systems incorporating embodiments of 

15 the novel imaging system 100, 122. Figure 1 illustrates one embodiment of an imaging 

system 100 coupled to a television 1 16 via a coax cable 114. The exemplifying system 
100 includes a lens 102, a sensor array 104 which may include one or more color filters, 
a readout control 106, gain amplifiers 108 for each color, an NTSC encoder 110 
including gamma correction and a power supply 112 used to power the camera circuitry 

20 and sensor array 104. The lens 102 may be used to focus images onto the sensor array 
104, and/or may be used as a protective covering for the sensor array 104. In one 
embodiment, the imaging system 100 is a video camera. In other embodiments, the 
imaging system 100 may be implemented in security cameras, digital cameras, 
camcorders, video telephones and the like. 

25 Figure 2 illustrates one embodiment of an imaging system 122 coupled to a 

computer 126 via a imiversal serial bus (USB) cable 124. The exemplifying system 
122 includes a lens 102, a sensor array 104 which may include one or more color filters, 
a readout control 106, gain ampUfiers 108 for each color, an analog to digital converter 
118, a USB interface 120 and a power supply 112 used to power the imaging system 

30 circuitry and sensor array 104. In one embodiment, the imaging system 122 is a video 
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camera. In other embodiments, the imaging system 122 may be similarly implemented 
in security cameras, digital cameras, camcorders, video telephones and the like. 

Figure 3 illustrates a conventional Red, Green and Blue (primary color system) 
Bayer color pattern 130 for a color filter that is deposited on an array of pixel cells that 
5 detect light. The pattern core is a group of 2 by 2 pixels that contains 2 green 

components 134, 136, one red component 132 and one blue component 138. The 2 
green components 134, 136 are diagonal neighbors, and the red and the blue 
components 132, 138 are diagonal neighbors. Thus, the green resolution of the array is 
reduced by a ratio of 2:1 horizontally only, while the red and the blue resolution is 

10 reduced by a ratio of 2:1 horizontally and vertically. The pattern 130 exploits the fact 
that the human eye perceives intensity edges better than color edges and that the green 
component contains the highest amount of intensity information. 

Figure 4 illustrates one embodiment of a circuit which utilizes the novel bad 
pixel detection and correction technique of the present invention. The illustrated 

15 embodiment is a CMOS integrated circuit, monochrome imaging system 198 that 

supports on-the-fly bad pixel detection & correction via programmable pixel readout 
circuitry 212, 214 and signal processing circuitry 200, 208. Other embodiments may 
employ CCD integrated circuit and/or different color filters. The signal processing unit 
may be analog or digital. The signal processing unit may be implemented using 

20 dedicated hardwired circuitry, a general purpose processor that may also perform other 
tasks, or a programmable circuit, such as a DSP, optimized for signal processing 
functions. 

As illustrated in Figure 4, the system 198 includes a bad pixel detection and 
correction unit 200, a pixel data output line 201, an analog-to-digital (A/D) converter 

25 208, an analog ampUfier 210, a column address decoder 212, a row address decoder 

214, a pixel address generator 216, a row coordinate line 226, a column coordinate line 
230, a frame synchronization signal 218, a line synchronization signal 220, a pixel clock 
signal 222, a control signal 224, a first column readout line 234, a second column 
readout line 236, a third column readout Une 238, a first switch 240, a first column 

30 buffer 246, a second switch 242, a second column buffer 248, a third column readout 

line 238, a third switch 244, a third column buffer 250, a first row readout line 262, a 
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second row readout line 264, a third row readout line 266, a first row buffer 252, a 
second row buffer 254, a third row buffer 256, a first pixel 258, a second pixel 260, a 
third pixel 266, a fourth pixel 268, and an optional storage unit 232. In one 
embodiment, the amplifier 210 is a programmable gain amplifier. 
5 Figure 4 presents an implementation for monochrome on-the-fly bad pixel 

detection & correction when windowing and sub-sampling is not active. The present 
invention, however, is not limited to a monochrome imager. Nor is the present 
invention Hmited in its mode of operation and can be realized to support window and 
sub-sampling via the programmable readout control circuitry 212, 214. In the 

10 description herein, a "programmable" component refers to a component that responds to 
a command firom an end-user of the imaging system or to a command issued by internal 
firmware according to firmware stored in the imaging system. For example, if an end- 
user chooses a 'zoom' fimction or a 'preview' fimction on a video camera containing the 
imaging system of the present invention, the imaging system directs the programmable 

15 components to act in a predefined manner according to firmware stored in the imaging 

system. 

In Figure 4, the system 198 contains a signal processing unit 200 that allows bad 
pixel detection and correction, as described below with reference to Figures 5A-B, The 
signal processing unit 200 treats the bad pixels as random *shot noise' and utilizes non- 
20 linear conditional filtering for detecting and eliminating or excluding the noisy or 
defective pixels. 

The imaging system of the present invention, such as the monochrome system 
198 in Figure 4, employs an on-the-fly bad pixel detection and correction process that is 
a signal processing procedure performed during the sensor readout stage. The system 

25 198 of Figure 4 may perform a random access readout of the pixels using the column 

and row decoders 212, 214. In on-the-fly bad pixel detection and correction, the signal 
processing unit 200 illustrated in Figure 4 performs bad pixel detection step via 
horizontal, vertical and diagonal conditional median filters within the signal processing 
unit 200. If a middle or center pixel is not within the condition limits, it is replaced in 

30 the correction step by a value determined by the directional median that possesses the 
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minimum variation, as described below. The value chosen can be either the median or 
an average. 

Figures 5A-B illustrate one embodiment of an on-the-fly bad pixel detection and 
correction process in fiill resolution mode implemented by the circuit of Figure 4. The 
5 present invention advantageously permits bad or defective pixels to be detected even 
after the imaging device has been provided to an end-user, such as a consumer, in a 
camera or the like. Thus, for example, the present invention allows bad pixel detection 
to be performed on-the-fly, in response to an end-user capturing an image using the 
camera, or during a calibration procedure automatically initiated when the user turns on 

10 the camera. 

Figure 6 illustrates an exemplifying 3x3 block of pixels within the system of 
Figure 4. In Figure 6, the block 300 comprises a first pixel 302, a second pixel 304, a 
third pixel 306, a fourth pixel 308, a fifth pixel 310, a sixth pixel 312, seventh pixel 314, 
an eighth pixel 316, and a ninth pixel 318 Figure 6 also shows a portion of a next 

15 column in the array of Figure 4. The next column comprises a tenth, eleventh and 

twelfth pixels 320, 322, 324. Figure 6 also shows a portion of the next row in the array 
of Figure 4. The next row comprises a thirteen, fourteenth and fifteenth pixels 326, 328, 
330. 

The process illustrated in Figures 5A-B is described with reference to Figures 4 
20 and 6. In a start state 502 of Figure 5A, the column and row readout circuits 212, 214 

illustrated in Figure 4A read out a 3 x 3 pixel block or neighborhood 300, such as that 
illustrated in Figure 6, of pixel sensor element values. The pixel values may be those 
for an image taken by an end-user using an imaging system, such as those illustrated in 
Figures 1 and 2. In a process state 504, the pixel address generator 216 illustrated in 
25 Figure 4 prepares four 1x3 sub-blocks comprised of pixels in a line: a middle 
horizontal sub-block, a middle vertical sub-block, and a first and second middle 
diagonal sub-blocks. The 1x3 sub-blocks may also be referred to as groups, local 
neighborhoods or subsets. The 1 x 3 middle horizontal sub-block comprises the fourth, 
fifth and sixth pixels 308, 310, 312. The middle vertical sub-block comprises the 
30 second, fifth and eighth pixels 304, 310, 316. The first middle diagonal sub-block 
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comprises the first, fifth and ninth pixels 302, 310, 318. The second middle diagonal 
pixel sub-block comprises the third, fifth and seventh pixels 306, 310, 314. 

In a process state 506, the bad pixel detection and correction unit 200 calculates 
the distribution for each 1x3 sub-block, i.e., the absolute difference between the 
5 highest and the lowest pixel sensor element values in a 1 x 3 sub-block. In a process 
state 508, the bad pixel detection and correction unit 200 finds the 1 x 3 sub-block with 
the minimum variance, i.e., the block with smallest absolute difference. In a process 
state 510, the bad pixel detection and correction unit 200 determines the median value 
for each 1x3 sub-block by sorting the three pixel values in each 1x3 sub-block from 

10 lowest to highest. For example, if a 1 x 3 block has values of 10, 15 and 100, the 
median value would be 15. 

In a decision block 512, the bad pixel detection and correction unit 200 
determines whether the value of the original middle pixel 310 illustrated in Figure 6 
exceeds the median values of the four 1x3 sub-blocks. If the original middle pixel 310 

15 value exceeds the median values of the four 1x3 sub-blocks, then in a process state 

514, the bad pixel detection and correction unit 200 replaces the original middle pixel 
310 value with the median value of the 1x3 sub-block with the minimum or smallest 
variance found in process state 508. Effectively, the bad pixel detection and correction 
unit 200 picks the smoothest edge. The bad pixel detection and correction unit 200 then 

20 passes the replaced middle pixel value via pixel data output line 201 for further 

processing, as is performed on the other pixels. In a process state 516, the pixel address 
generator 216 and the bad pixel detection and correction unit 200, illustrated in Figure 
4, advance to the next column, such that the middle pixel is now the sixth pixel 312 
illustrated in Figure 6. 

25 If the unit 200 determines that the original middle pixel 310 value does not 

exceed the median values of the four 1x3 sub-blocks, then in process state 516, the 
pixel address generator 216 and bad pixel detection and correction unit 200 advance to 
the next column, such that the middle pixel is now the sixth pixel 312 in Figure 6. In a 
decision block 366, the pixel address generator 216 determines whether it has exceeded 

30 the last column in the pixel array of Figure 4. If the pixel address generator 216 has not 



-12- 



exceeded the last column in the pixel array, the generator 216 and the bad pixel 
detection and correction unit 200 begin to process the next 3x3 block. 

If the pixel address generator 216 has exceeded the last column in the pixel array 
of Figure 4, then in a process state 520, the generator 216 advances to the next row of 
5 pixel sensor elements, such that the middle pixel is now the eighth pixel 316, In a 
decision state 522, the address generator 216 determines whether it has exceeded the 
last row in the array of Figure 4, If the address generator 216 has not exceeded the last 
row in the array, the generator 216 and the bad pixel detection and correction unit 200 
begin to process the next 3x3 block. If the address generator 216 has exceeded the last 
10 row in the array, the generator stops in a stop block 524. 

The bad pixel detection and correction process may also be described via the 
following exemplary mathematical algorithm which may be implemented in software or 
hardware. 

In this example, the pixel neighborhood used is a 3 x 3 group as follows: 

15 

^(0,0)' ^(0,1 > ^(0,2) 
^(1,0>^(1,1>^(1,2) 
^(2,0)'^(2,1>^(2,2) 

20 Then, using the digitized pixel values, the median values for the four 3 -pixel subsets 

that include the middle pixel are calculated, and the maximum and minimum 
median values are assigned correspondingly to the variables Max and Min as follows: 

(1) Max = Maximum [ Median {X^\fiy -^(1,2) )? Median (X(oj)? ^(ij)? ^(2,i) )' 
25 Median (X(o^o> ^(i,i> %2) \ Median (X^q 2), X(i,i). X(2,o) ) 1 

(2) Min = Minimum [ Median {X^i^^y X^^^^y 2) ), Median (X(oj), X^^^^y X^2^) )^ 

Median (X^q^q), j), X(2^2) )^ Median (X(o^2)^ ^(2,o) ) 1 

30 (3) The ratios of the middle pixel value to the Max value and to the Min value are 

compared to a threshold value. The threshold value is a predetermined value so 
chosen that if the above ratios are greater than the threshold, there is a 
statistically high likelihood that the middle pixel value is in error. 



Limit = 



^ Y ^ 
> Threshold 
^ Max J 



( \ 
Min „ _ , ^ 
or > Threshold 



35 
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(4) If ( Limit > 0) then the value of the middle pixel will be replaced with the 
median value of one of the four 3-pixel subsets. The median value of the subset 
with the smallest variance between the values of the most physically distant 
pixels within the subset will be used to replace the middle pixel value, as 
5 follows: 

(5) Variance = Minimum ( ^(^^o) - ^(i,2) I l^(0,i) " %i) I' l^(0,0) " ^(2,2) I? 

1^(0,2) " ^(2,0) I ) 

10 (6) If Variance e {X^i QyX^i iy X^i 2)} Then 

(7) = Median (X(i 
(8) If Variances {X^QiyX^i^iyX(2^i^} Then 

15 

(9) = Median (Zfo,i> %i)) 
(10) If Variance e {X^(^ QyX^i iyX^2,2)} Then 
20 (11) = Median (X(o,o> ^(U). %2)) 

(12) If Variance e {X^q 2)3^(1, i)?^(2,o)} Then 

(13) = Median {X^^^^y ^(i,i> ^(2,0)) 

25 

Figure 9 illustrates an exemplifying circuit used to locate a minimum pixel 
value, an intermediate pixel value, and a maximum pixel value in a group of three 
pixels, such as one of the groups used in the above equations. This circuit may be 
30 implemented in a gate array, a field programmable gate array, a custom integrated 
circuit, using discrete logic, or in software. Optionally, the illustrated circuit may be 
repeated four times so that the pixels having the minimum, maximum, and intermediate 

value for each of the pixel groups (X^i o), X^x^y ^(1,2) ). (^(o,i). X^uy ^(2,1) \ (^(o,o). ^(i.i). 
^(2,2) )> (^(0,2)' ^(1,1)' ^(2,0) ) i^^y be located in parallel, thereby speeding the process. In 
35 another embodiment, to reduce circuitry, the location process may be serially performed 

using only one circuit. 

In the illustrated example, pixel 1 corresponds to X(i^o), pi^^l 5 corresponds to 
X^i^iy and pixel 9 corresponds to X^y 2y The corresponding signs sgnA, sgnB, sgnC, of 
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the values of (pixel 1 - pixel 5), (pixel 5 - pixel 9), and (pixel 1 - pixel 9) are calculated. 

The signs of the results are used to address a look-up table. A positive sign is a "0" and 

a negative sign is a "1" for purposes of addressing. The lookup table maps the sign 

values into an identification of the pixel having the minimum value, the pixel having the 

5 maximum value, and the pixel having the minimum value. Thus, for example, if pixel 1 

has a normalized value of 0.9, pixel 5 has a normahzed value of 0.7, and pixel 9 has a 

normalized value of 0.8, then 

sgnA is positive, or a "0" 
sgnB is negative, or a "1" 
10 sgnC is positive, or a "0" 

Thus, the address is a 010. The location information at the location addressed by 010 

identifies pixel 5 as having the minimum value, pixel value having a middle or 

intermediate value, and pixel 1 as having a maximum value. This location information 

may be then be used in calculating the distribution for each 1x3 sub-block, such as the 

15 difference between the highest and the lowest pixel element sensor value calculated at 

process state 506 in Figure 5. A similar circuit may be used to locate the highest, 
intermediate, and lowest median value fi'om a group of median values, as performed in 
Equations 1 and 2 above. 

It is important to note that the present invention is not limited to the 

20 embodiments discussed above. Other configurations may be implemented in 
accordance with the present invention. For example, in one embodiment, instead of or 
in addition to using straight 1x3 diagonal pixel sub-blocks, the address generator 216 
and bad pixel detection and correction unit 200 may use a semi-diagonal 1x3 sub- 
block, such as the third, fifth and eighth pixels 306, 310, 316 of the block illustrated in 

25 Figure 6. In another embodiment, the address generator 216 may use a semi-diagonal 1 
x 3 sub-block, such as the third, fifth and ninth pixels 306, 310, 318, 

In still another embodiment, instead of or in addition to using the various 1x3 
pixel sub-blocks discussed above, the address generator 216 and the bad pixel detection 
and correction unit 200 may use the complete 3x3 pixel neighborhood for one median 

30 filtering of the 9 elements. In still another exemplary embodiment, the mean values for 
all the blocks in the neighborhood are utiUzed, rather than median values. 
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In other embodiments, the bad pixel correction does not rely on minimum 
variance. For example, any of the median values can be chosen arbitrarily as the bad 
pixel replacement. Further, other values, such as the average values of some or all of 
the pixels in a block or group may be used. 
5 In another embodiment, the address generator 216 and bad pixel detection and 

correction unit 200 may be configured to use other sizes of blocks or groups of pixels. 
For example, a 5 x 5 pixel block instead of a 3 x 3 block may be used, such that each 
median is selected from a 1 x 5 pixel sub-block instead of a 1 x 3 sub-block. In one 
embodiment, the pixel to be examined may be included in each 1x5 pixel sub-block. 

10 Rather than examining or testing the value of a center pixel within the 1 x 5 pixel sub- 
block, other pixels, such as other interior pixels having a pixels from the block 
positioned on two sides, may be used. Other embodiments may detect and correct pixel 
values using the values of pixels in a two dimensional neighborhood having other 
geometric shapes^ such as circular or triangularly shaped neighborhoods, or using 

15 neighborhoods having irregular shapes. In still another embodiment, rather than using 

only pixels within a neighborhood in detecting and correcting pixel failures or 
misreadings for pixels within the neighborhood, all array pixels may be used, with more 
weight being given to those pixels proximate to a pixel whose value is being inspected 
and/or corrected. 

20 Furthermore, for color imaging systems, a block with color fiher components 

may be used. For example, a color imaging system with an RGB Bayer pattem may use 
a 5 X 5 pixel neighborhood out of which only pixels of the same color component as the 
middle pixel are used for the filtering process. Figure 7 illustrates an exemplifying 5 x 
5 sub-block of pixels using the primary color scheme and the Bayer color pattem where 

25 the middle pixel 360 is red. In Figure 7, the pixel address generator 216 and the bad 

pixel detection and correction unit 200 applies the same general process shown in 
Figures 5A-B, except that only the red pixel values 352-368 are used in the 1 x 5 pixel 
sub-blocks. 

Figure 8 illustrates an exempHfying 5x5 block of pixels using the primary color 
30 scheme and the Bayer color pattem, where the middle pixel 390 is blue. In Figure 8, the 

pixel address generator 216 and the bad pixel detection and correction unit 200 appUes 
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the same general process shovm in Figure 5A-B, except that only the blue pixel values 
382-398 are used in the 1 x 5 pixel sub-blocks. 

In one embodiment, the imaging system of the present invention uses random 
access readout. In other embodiments, the imaging system uses different 
5 implementations such as a pipeline architecture or a parallel readout architecture. A 
pipeline architecture may use a sequential readout via row and column shift registers 
and several line storage units, such as random access memory (RAM) in digital domain 
or capacitors in analog domain. This implementation may reduce the complexity of 
control circuitry, but may require additional memory or storage elements as compared 

10 to the embodiment illustrated in Figure 4. For example, referring to Figure 4, a RAM or 
3 pixel register delay circuit may be inserted before the bad pixel detection and 
correction unit 200. The column address decoder may optionally be replaced with a 
simpler column shift register clocked by the line sync signal Similarly, the row address 
decoder may optionally be replaced with a row shift register clocked by a frame sync 

15 signal. In a parallel readout architecture, several pixels per column can be read 

simultaneously via independent column buses and stored in registers. 

In other embodiments, the imaging system fiirther combines the on-the-fly bad 
pixel detection & correction with various readout modes such as windowing and sub- 
sampling. In one embodiment, the imaging system uses additional signal processing 

20 such as on-the-fly color interpolation. 

As previously discusses, the pixels of the imaging system are not required to be 
organized in a rectangular matrix. A similar implementation can provide on-the-fly bad 
pixel correction and detection for a color imaging system and/or a different pixel 
topography via modification to the readout control, the pixel neighborhood size and 

25 configuration. 

Thus, as described above, the various embodiments of the present invention 
advantageously provides an on-line, on the fly defective pixel detection and correction 
process that provides high speed operation, reduced storage requirements, and improved 
detection of defective pixels as compared to conventional systems. 

30 The invention may be embodied in other specific forms without departing from 

its spirit or essential characteristics. The described embodiment is to be considered in 
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all respects only as illustrative and not restrictive and the scope of the invention is, 
therefore, indicated by the appended claims rather than the foregoing description. All 
changes which come within the meaning and range of equivalency of the claims are to 
be embraced within their scope. 
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WHAT IS CLAIMED IS : 

. An imaging system configured to compensate for one or more individual 
defective pixels in an array of pixel elements, the system comprising: 
an array of pixel sensor elements; 
5 a readout controller coupled to the array, the readout controller 

configured to read a block of pixels within the array, the block including a center 
pixel, the controller further configured to generate pixel addresses of at least 
three subsets of pixels within the block; and 

a bad pixel detection and correction unit configured to determine a subset 
10 with a minimum variance, calculate a median of each subset, determine whether 

a value of the center pixel exceeds the medians of the subsets, and to replace the 
value of the center pixel with the median of the subset with the minimum 
variance if the value of the center pixel exceeds the medians of the subsets. 

2. The imaging system as defined in Claim 1, wherein the array comprises a 
1 5 monochrome array. 

3. The imaging system as defined in Claim 1, wherein the system fiirther 
comprises a color filter deposited on the array. 

4. The imaging system as defined in Claim 1 , wherein the subsets comprise 
a horizontal row and two diagonal patterns. 

20 5. The imaging system as defined in Clahn 1, wherein the subsets comprise 

a vertical row and two diagonal pattems. 

^6. A method of correcting one or more individual defective pixels in an 
array of pixel elements, the method comprising: 

reading a pixel block of an array of pixel sensor elements, the pixel block 
25 including an interior pixel; 

selecting at least a first subset, a second subset and a third subset of 
pixels within the pixel block, the first subset comprising pixels arranges in a first 
diagonal pattern within the block, the second subset arranged in a second 
diagonal pattern within the block, the third subset arranged in a non-diagonal 
30 pattem; 
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calculating a variance between a highest and a lowest pixel sensor value 
within each subset; 

identifying a subset with the least variance; 

calculating a first value based on at least one of a median, average, and 
5 mean pixel sensor value for the corresponding three subsets; 

determining whether a value for the interior pixel varies from the first 
value by more than a first amount; and 

replacing the value for the interior pixel with a replacement value based 
on the pixel sensor values of the subset with the lowest variance if the value for 
10 the interior pixel varies from the first value by more than a first amount. 

7. The method of Claim 6, further comprising: 

selecting a fourth subset arranged in another non-diagonal pattern; and 
calculating a variance between a highest and a lowest pixel sensor value 
within the fourth subset. 
15 8. The method of Claim 6, wherein the non-diagonal pattern is horizontal. 

9. The method of Claim 6, wherein the non-diagonal pattem is vertical. 

10. The method of Claim 6, wherein the act of reading a pixel block 
comprises reading a 3 x 3 pixel block, 

11. The method of Claim 6, wherein the act of reading a pixel block 
20 comprises reading a 5 x 5 pixel block. 

V2^ A method of detecting a defective pixel element within an array of pixel 
elements in an imaging device while the imaging device is in use by an end-user, the 
method comprising: 

capturing an image taken by the end-user using the imaging device; 
25 for the captured image, comparing a first pixel element value with a 

second value related to at least element values of other imaging pixel elements 
in a first group; 

determining from the comparison if the first pixel element value is in 
error; and 
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substituting a third value related to the value of at least one of the other 
pixels elements at least partly in response to determining the first pixel element 
value is error. 

13. The method as defined in Claim 12, wherein the comparison act includes 
5 comparing the value of the first pixel element with the median value of the first group of 

pixels. 

14. The method as defined in Claim 12, wherein the first group include at 
least two pixel elements adjacent to the first pixel element. 

15. The method as defined in Claim 12, wherein the second value is also 
10 related to the first pixel element value, 

16. The method as defined in Claim 12, wherein the third value is related to 
the median value of at least two other pixel values. 

17. The method as defined in Claim 12, wherein the imaging device is a 
color imaging device, and the other pixels whose values are compared to the first pixel 

15 value are intended to sense the same color as the first pixel element. 

18. The method as defined in Claim 12, wherein the imaging device is a 
monochrome imaging device. 

1^ An imaging system configured to compensate for one or more individual 
defective pixels in an imaging array, the system comprising: 
20 a readout controller coupled to the imaging array, the readout controller 

configured to read a group of pixels within the array; 

a defective pixel detection circuit configured to determine when at least a 
first pixel value associated with a first pixel within the group of pixels varies 
fi-om a second value related to at least one other pixel within the group of pixels 
25 by a first amount; and 

a pixel compensation circuit configiu*ed to replace the value of the first 
pixel with a third value related to at least one other pixel within the group of 
pixels when the first pixel value varies by more than the first amount fi:om the 
second value. 

30 20. The imaging system as defined in Claim 19, wherein the second value is 

a median value of a plurality pixel values of pixels within the group of pixels. 
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21. The imaging system as defined in Claim 19, wherein the second value is 
an average value of a plurality pixel values of pixels within the group of pixels. 

22. The imaging system as defined in Claim 19, wherein the third value is 
related to the median value of a plurality pixel values of pixels within the group of 

5 pixels. 

23. The imaging system as defined in Claim 19, fiirther comprising a color 
filter overlaying at least a portion of the array. 

24. The imaging system as defined in Claim 19, wherein the array is a 
CMOS array. 

10 25. The imaging system as defined in Claim 19, wherein the array is a CCD 

array. / 

^6. A camera system, comprising: 

an imager, including a plurality of pixels; 
a lens overlaying at least a portion of the imager; 
15 a readout circuit coupled to the imager, the readout circuit configured to 

read imager pixel values; 

a defective pixel detection circuit configured to determine if a first pixel 
is defective by examining the pixel values of a plurality of pixels readout by the 
readout circuit; 

20 a pixel compensation circuit configured to substitute the value of the first 

pixel with a value related to at least one other pixel value readout by the readout 
circuit; and 

a power supply used to power the readout controller, the defective pixel 
detection circuit, and the pixel compensation circuit. 
25 27. The camera system as defined in Claim 26, wherein the camera system is 

a video camera. 

28. The camera system as defined in Claim 26, fiirther comprising an NTSC 
encoder coupled to the readout circuit. 

29. The camera system as defined in Claim 26, wherein the camera system is 
30 located on a phone. 
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30. The camera system as defined in Claim 26, further comprising a color 
filter positioned over at least a portion of the imager. 

l>y, A method of detecting a defective pixel element within pixel elements in 
an imaging device using an image captured by the end-user, the method comprising: 
5 for the captured image, comparing a first pixel element value with a 

second value related to at least element values of pixel elements in a two 
dimensional neighborhood; 

determining from the comparison if the first pixel element value is in 
error; and 

10 substituting the first pixel element value with a third value related to a 

value of at least one of the other pixels elements in the two dimensional 
neighborhood. 

32. The method as defined in Claim 31, wherein the second value is related 
to at least two element values corresponding to at least two pixels on opposite sides of 

1 5 the first pixel element. 

33. The method as defined in Claim 31, wherein the comparison includes 
determining if the first pixel element value varies from the second value by more than a 
threshold amount. 

3^, An imaging system comprising: 
20 an imager including a plurality of pixel sensor elements; 

a controller coupled to the imager, the controller configured to read pixel 
sensor element values; 

a defective pixel detection circuit configured to determine when at least a 
first pixel value associated with a first pixel sensor element within a two 
25 dimensional neighborhood is in error by comparing the at least first pixel value 

to a second value related to at least one other pixel element within the two 
dimensional neighborhood; and 

a pixel compensation circuit configured to replace the value of the first 
pixel element with a third value related to at least one other pixel element within 
30 the two dimensional neighborhood. 
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METHODS AND SYSTEMS OF DETECTING DEFECTIVE IMAGING 
PIXELS AND PIXEL VALUES 



Abstract of the Disclosure 

5 The present invention is related to methods and systems for detecting defective 

imaging array pixels and providing correction, thereby reducing or eliminating visible 
image artifacts. One embodiment of the present invention provides an on-line bad pixel 
detection and correction process that compares a first pixel readout value with a first 
value related to the readout values of other pixels in first pixel's local neighborhood. 

10 When the first pixel readout value varies by more than a first amount as compared with 
the first value, a second value related to the readout values of the neighboring pixels is 
used in place of the first pixel readout value. 

15 
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